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INTRODUCTION 

Are U.S. and, in particular, Hoosier stu- 
dents competitive and ready to succeed in 
an ever-changing and increasingly global 
economic landscape? This question is 
frequently considered by K-12 education 
stakeholders at all levels, including nation- 
al, state, and local officials. For example, 
President Obama contends that “providing 
a high-quality education for all children 
is critical to America’s economic future. 
Our nation’s economic competitiveness... 
depend[s] on providing every child with an 
education that will enable them to succeed 
in a global economy” (The White House, 
n.d.). Within Indiana, Tony Bennett, for- 
mer State Superintendent of Schools, used 
increasing economic competitiveness 
as a cornerstone of the state’s education 
reforms, stating that secondary and pri- 
mary education partnerships with private 
industry and universities “will help Indi- 
ana develop the nation’s best workforce, 
ready to tackle the challenges of our global 
economy” (University of Indianapolis, 
2012). At the local level, school boards 
have also supported these perspectives, 
with the school board of Indianapolis Pub- 
lic Schools stating its vision for education 
as “innovative urban education, preparing 
all students to be successful in the global 
economy” (Indianapolis Public Schools, 
2011). One of the central ways in which 
education systems can compare them- 
selves internationally is through regularly 
administered education assessments, such 
as the Trends in International Mathematics 
and Science Study (TIMSS) or the Pro- 
gramme for International Student Assess- 
ment (PISA). 

Both PISA and TIMSS are large-scale in- 
ternational assessments for students, while 


the National Assessment of Educational 
Progress (NAEP), also referred to as the 
“Nation’s Report Card,” is designed to gath- 
er information that meets national and state 
needs. The important distinction here is that 
NAEP is a national assessment, and PISA 
and TIMSS are international assessments. 

The primary difference between PISA and 
TIMSS is what the assessments intend to 
measure. For example, PISA is concerned 
with how students are able to apply their 
knowledge to real life situations in read- 
ing, mathematics, and science. PISA is 
administered every three years to a sam- 
ple of 15-year-old students who are near- 
ing completion of compulsory schooling. 
TIMSS focuses on how students are learn- 
ing specific material from an internation- 
ally agreed upon curriculum in mathemat- 
ics and science, and is administered every 
four years to student samples from the 4 th 
and 8 th grades, in order to assess learning at 
different stages. 

Historically, the U.S. ranks consistently in 
the middle of the pack among national par- 
ticipants in most international education 
assessments. Given that the U.S. is a large 
country with diverse state and local educa- 
tion systems that can and do vary meaning- 
fully, aggregating and reporting at the na- 
tional level provides useful information for 
national policymakers, but does little for 
state and local policymakers. Fortunately, 
in 2011 Indiana participated in the latest 
round of 8 th grade TIMSS assessments. 1 

To that end, this policy brief examines the 
most recent TIMSS results for Indiana stu- 
dents in order to compare Hoosier 8 th grad- 


1 Although Indiana participated in TIMSS at both 
the 4 th and 8 th grade in 2003, Indiana 4 th graders 
did not participate in TIMSS 2011. Further, 
TIMSS only assessed 8 th grade students in 1999. 


ers with their global peers, looking at aver- 
ages for the Top 10 performing countries 
(averaging education systems at the coun- 
try level, excluding the state of Indiana), 
the U.S., and the world. This brief will 
present the results in terms of system-level 
populations. Disaggregated results by gen- 
der and system-level comparisons across 
TIMSS benchmarks are also presented and 
discussed. The brief will conclude with an 
examination of achievement trends in Indi- 
ana and internationally from 1999 to 2011, 
including a short discussion of what these 
results mean in the context of current edu- 
cation reforms in the state. 


TIMSS RESULTS: A FOCUS ON 
INDIANA STUDENTS 

Similar to how Indiana 8 th grade students 
performed in 2003, Indiana’s 2011 TIMSS 
performance was above the U.S. average 
for 8 th grade students in both mathematics 
and science (Chien, Spradlin, & Plucker, 
2007; Martin, Mullis, Foy, & Stanco, 
2012; Mullis, Martin, Foy, & Arora, 2012). 
Table 1 shows that only six education 
systems around the world performed sta- 
tistically significantly better in mathemat- 
ics than Indiana. An important point here 
is that Indiana outperformed Finland, a 
country widely recognized in academic 
literature and popular media for its the 
success of its education system. Although 
Indiana’s rank in science was lower than 
in mathematics, the state nonetheless per- 
formed well internationally in science, 
with only five education systems outper- 
forming Hoosier 8 th graders (see Table 
2). In addition, Indiana’s performance 
was not statistically significantly differ- 
ent from Hong Kong in science, which 
is typically a top-performer on a range 
of international education assessments. 

Figure 1 represents Indiana’s achieve- 
ment in both mathematics and science in 
comparison to the international average, 
the U.S. average, and the average achieve- 
ment of the Top 10 performing countries 
(excludes Indiana). From this graphic 
representation, several interesting find- 
ings emerge. First, Indiana has stronger 
performance in science than mathematics; 
this trend is consistent nationally and inter- 
nationally; however, the opposite is true of 
the Top 10 countries, which excludes the 
state of Indiana. That is, the top perform- 


ers have higher mathematics than science 
achievement. Next, Indiana’s mathematics 
and science achievement is lower than the 
Top 10 countries, which, again, excludes 
the state of Indiana; however, Hoosier 8 th 
graders perform above the international 
average in both mathematics and science. 
Further, Indiana’s mathematics achieve- 
ment is stronger than the rest of the U.S. 
and its science performance is on par with 
other U.S. students. Given the uncertainty 
around these achievement estimates (rep- 
resented by the black brackets, described 
in the note below Figure 1), we can also 
see that Indiana is not far behind the high- 
est performing countries in both math- 
ematics and science; however, certain 
weaknesses should be addressed if Indiana 
aims to compete globally in terms of sec- 
ondary education. 


Gender Differences in Achievement 

In both the U.S. and Indiana there has been 
a large emphasis on closing gender gaps 
in both mathematics and science. Overall 
no statistically significant math differences 
exist between boys and girls (see Table 3); 
however, a statistically significant gap in 
science exists nationally (see Table 4). Giv- 
en no gender differences in mathematics 
but a statistically significant gap favoring 
U.S. boys in science, the evidence is mixed 
regarding whether efforts to close the gen- 
der gap are working at the national levels. 


Unfortunately, a more consistent picture 
emerges in Indiana. In both mathematics 
and science, a statistically significant gen- 
der gap favoring boys is present. Given 
a marked underrepresentation of women 
in science, technology, and engineering 
fields, both nationally and within Indiana, 
addressing these gender gaps could be an 
important step to improving economic 
competitiveness for Indiana in these areas. 

Figure 2 further illustrates the discrepan- 
cies between boys and girls in mathemat- 
ics achievement. Both internationally and 
among the Top 10 performers, small, non- 
significant gender gaps favoring girls ex- 
ist. As previously noted, the opposite is 
true both nationally and in Indiana. That 
is, gender gaps tend to favor boys in math- 
ematics and this gap is statistically sig- 
nificant in Indiana. This troubling finding 
suggests that further investigation in the 
Indiana context is warranted to ensure that 
boys and girls have equal opportunities to 
learn and succeed in these subject areas. 

Figure 3 shows that the Top 10 performing 
countries have a small achievement gap in 
science that favors boys, while the interna- 
tional average score favors girls. Notable in 
this figure is the boys’ performance in Indi- 
ana as compared to the Top 10 performing 
countries. Indiana boys perform nearly as 
well as their Top 10 counterparts when the 
uncertainty around achievement estimates 
is taken into account. Despite this posi- 
tive finding, the science achievement gap 


Figure 1. TIMSS 2011 Comparison of Mathematics and Science Average Scores 



Note 1: The scale scores begin at zero, but no education system scored below 300 (Martin et al., 2012; 
Mullis et al., 2012). 

Note 2: The brackets at the end of each bar represent plus and minus two standard errors. 

Note 3: The 2011 TIMSS data did not provide standard errors for the international average, nor were 
standard errors calculated for the Top 10 performing countries averaged score. 
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Table 1. TIMSS 2011 Comparison of Grade 8 Mathematics Ave. Scores 


Table 2 TIMSS 2011 Comparison of Grade 8 Science Average Scores 


Education System 

Score 

Significantly different from 
Indiana?** 

South Korea 

613 

Yes 

Singapore 

611 

Yes 

Chinese Taipei (China) 

609 

Yes 

Hong Kong (China) 

586 

Yes 

Japan 

570 

Yes 

Russian Federation 

539 

Yes 

Indiana 

522 

- 

Israel 

516 

No 

Finland 

514 

No 

United States 

509 

Yes 

England 

507 

No* 

Hungary 

505 

Yes 

Australia 

505 

Yes 

Slovenia 

505 

Yes 

Lithuania 

502 

Yes 

Italy 

498 

Yes 

New Zealand 

488 

Yes 

Kazakhstan 

487 

Yes 

Sweden 

484 

Yes 

Ukraine 

479 

Yes 

Norway 

475 

Yes 

Armenia 

467 

Yes 

International Average 

467 

- 

Romania 

458 

Yes 

United Arab Emirates 

456 

Yes 

Turkey 

452 

Yes 

Lebanon 

449 

Yes 

Malaysia 

440 

Yes 

Georgia 

431 

Yes 

Thailand 

427 

Yes 

Macedonia 

426 

Yes 

Tunisia 

425 

Yes 

Chile 

416 

Yes 

Iran 

415 

Yes 

Qatar 

410 

Yes 

Bahrain 

409 

Yes 

Jordan 

406 

Yes 

Palestinian Nat'l Auth. 

404 

Yes 

Saudi Arabia 

394 

Yes 

Indonesia 

386 

Yes 

Syrian Arab Republic 

380 

Yes 

Morocco 

371 

Yes 

Oman 

366 

Yes 

Ghana 

331 

Yes 


* This seemingly anomalous finding is due to the measures of uncertainty around each 
parameter estimate. That is, the measure of uncertainty around England's estimate is 
wider than that of the U.S., which leads to a non-significant difference between Indiana 
and England and a significant difference between Indiana and the U.S. 

**Statistical differences were estimated using Bonferroni corrections at an unadjusted 
Type I error rate of .05. 


Education System 

Score 

Significantly different from 
Indiana?* 

Singapore 

590 

Yes 

Chinese Taipei (China) 

564 

Yes 

South Korea 

560 

Yes 

Japan 

558 

Yes 

Finland 

552 

Yes 

Slovenia 

543 

No 

Russian Federation 

542 

No 

Hong Kong (China) 

535 

No 

England 

533 

No 

Indiana 

533 

- 

United States 

525 

No 

Hungary 

522 

No 

Australia 

519 

Yes 

Israel 

516 

Yes 

Lithuania 

514 

Yes 

New Zealand 

512 

Yes 

Sweden 

509 

Yes 

Italy 

501 

Yes 

Ukraine 

501 

Yes 

Norway 

494 

Yes 

Kazakhstan 

490 

Yes 

Turkey 

483 

Yes 

International Average 

477 

- 

Iran 

474 

Yes 

Romania 

465 

Yes 

United Arab Emirates 

465 

Yes 

Chile 

461 

Yes 

Bahrain 

452 

Yes 

Thailand 

451 

Yes 

Jordan 

449 

Yes 

Tunisia 

439 

Yes 

Armenia 

437 

Yes 

Saudi Arabia 

436 

Yes 

Malaysia 

426 

Yes 

Syrian Arab Republic 

426 

Yes 

Palestinian Nat'l Auth. 

420 

Yes 

Georgia 

420 

Yes 

Oman 

420 

Yes 

Qatar 

419 

Yes 

Macedonia 

407 

Yes 

Lebanon 

406 

Yes 

Indonesia 

406 

Yes 

Morocco 

376 

Yes 

Ghana 

306 

Yes 


^Statistical differences were estimated using Bonferroni corrections at an unadjusted 
Type I error rate of .05. 
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between Hoosier boys and girls points to 
a need for efforts that will level the learn- 
ing field in science for girls. Further, given 
that gender gaps exist in both mathematics 
and science, it might be sensible to also ask 
whether this gap extends into other learn- 
ing areas for girls in Indiana. 


COMPARISON TO TIMSS 
ACHIEVEMENT BENCHMARKS 

As an additional means for comparing 
achievement, the TIMSS project devel- 
oped four achievement benchmarks: ad- 
vanced, high, intermediate, and low. These 
benchmarks provide a useful context for 
interpreting the meaning behind achieve- 
ment scale scores (see Table 5 for an ex- 
planation of the benchmarks). From Figure 
4, we see that Indiana has a higher percent- 
age of students reaching the high level of 


performance in mathematics than in the 
U.S. as a whole, while it has the same per- 
centage of students reaching the advanced 
level. Unfortunately, both the U.S. and In- 
diana fail to produce a high percentage of 
students in the advanced level, particularly 
in comparison to the Top 10 international 
performers. Figure 5 displays results from 
science, which are similar to the math- 
ematics scores. From these data it appears 
that the U.S. and Indiana consistently 
struggle to produce significant numbers of 
advanced-level achievers, which is a nota- 
ble weakness in both the U.S. and Indiana 
education systems. 


Table 3. TIMSS 2011 Difference in Mathematics Average Score Between Genders 


Education System 

Girls 

Boys 

Difference 

(+/-)! 

Significant 

Difference* 

Korea, Rep. of 

610 

616 

-6 

Yes 

Singapore 

615 

607 

+8 

Yes 

Chinese Taipei (China) 

613 

606 

+7 

No 

Hong Kong (China) 

588 

583 

+5 

No 

Japan 

566 

574 

-8 

No 

Russian Federation 

539 

539 

0 

- 

Indiana 

518 

526 

-8 

Yes 

Israel 

520 

512 

+8 

No 

Finland 

516 

512 

+4 

No 

United States 

508 

511 

-3 

No 

England 

508 

505 

+3 

No 

International Average 

469 

465 

+4 

- 


Note 1: A positive difference favors girls while a negative difference favors boys. 

Note 2: Only the top ten performers, Indiana, and the international average are provided for comparison. 
^Statistical differences were estimated using Bonferroni corrections at an unadjusted Type I error rate of .05. 


Table 4: TIMSS 2011 Difference in Science Average Score Between Genders 


Education System 

Girls 

Boys 

Difference 

(+/-)! 

Significant 

Difference* 

Singapore 

589 

591 

-2 

No 

Chinese Taipei (China) 

564 

564 

0 

- 

Korea, Rep. of 

558 

563 

-5 

No 

Japan 

554 

562 

-8 

Yes 

Finland 

555 

550 

+5 

No 

Slovenia 

541 

545 

-4 

No 

Russian Federation 

539 

546 

-7 

Yes 

Hong Kong (China) 

536 

534 

+2 

No 

England 

534 

532 

+2 

No 

Indiana 

526 

541 

-15 

Yes 

United States 

519 

530 

-11 

Yes 

International Average 

480 

474 

+6 

- 


Note 1: A positive difference favors girls while a negative difference favors boys. 

Note 2: Only the top ten performers, Indiana, and the international average are provided for comparison. 
^Statistical differences were estimated using Bonferroni corrections at an unadjusted Type I error rate of .05. 
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Figure 2. TIMSS 2011 Comparison of Mathematics Average Scores Between Genders 
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Note 1: The scale scores begin at zero, but no education system scored below 300 (Martin et al., 
2012; Mullisetal., 2012). 

Note 2: The brackets at the end of each bar represent plus and minus two standard errors. 

Note 3: The 2011 TIMSS data did not provide standard errors for the international average, nor 
were standard errors calculated for the Top 10 performing countries averaged score. 


Figure 3. TIMSS 2011 Comparison of Science Average Scores Between Genders 
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Note 1: The scale scores begin at zero, but no education system scored below 300 (Martin et al., 
2012; Mullisetal., 2012). 

Note 2: The brackets at the end of each bar represent plus and minus two standard errors. 

Note 3: The 2011 TIMSS data did not provide standard errors for the international average, nor 
were standard errors calculated for the Top 10 performing countries averaged score. 


ACHIEVEMENT TRENDS 

This policy brief has provided informa- 
tion concerning the 2011 TIMSS in which 
Indiana 8th graders participated. It has 
revealed the importance of comparison to 
groups of other education systems and sev- 
eral key policy areas that can be addressed 
by disaggregating results by gender and 
by examining benchmarks for student 
achievement. 

As a final comparison, we present the 
trends in mathematics (Figure 6) and sci- 
ence (Figure 7) for the years in which Indi- 
ana’s 8 th grade students have participated. 
It appears that Indiana struggled with the 
2003 TIMSS assessments, as evidenced by 
a dip in achievement when compared to 
1999 and 2011 performance in both math- 
ematics and science. This could be due to a 
variety of reasons, from the test being a bit 
more difficult to actual changes in student 
performance; however, given that the ma- 
jority of countries saw lower performance 
in 2003 — which stabilized over time — the 
deviation does not appear to merit policy 
concern. 


Table 5. TIMSS Benchmark Definitions 


Level 

Score 

Mathematics 

Science 

Advanced 

625 or above 

Reason, draw conclusions, make generaliza- 
tions, and solve linear equations 

Communicate an understanding of complex 
and abstract concepts in biology, chemistry, 
physics, and earth science 

High 

550-624 

Apply knowledge and understanding in a 
variety of relatively complex situations 

Demonstrate understanding of concepts 
related to science cycles, systems, and 
principles 

Intermediate 

475-549 

Apply basic knowledge in a variety of situa- 
tions 

Apply understanding of basic scientific 
knowledge in various contexts 

Low 

474 or below 

Some knowledge of whole numbers and 
decimals, operations, and basic graphs 

Recognize some basic facts from the life and 
physical sciences 


Sources: Mullis et al. (2012, p. 8) and Martin et al. (2012, p. 8). 
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Figure 4. TIMSS 2011 Comparison of Mathematics International 
Benchmarks 


Figure 5. Comparison of Science International Benchmarks 
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Figure 6. TIMSS 2011 Trends in Performance for Mathematics 


Figure 7. TIMSS 2011 Trends in Performance for Science 
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Sources: Mullis et al. (2012). 

Note 1: The brackets it juncture point represent plus and minus two stan- 
dard errors. 

Note 2: The 2011 TIMSS data did not provide standard errors for the 
international average, nor were standard errors calculated for the Top 10 
performing countries averaged score. 


Sources: Martin et al. (2012). 

Note 1: The brackets it juncture point represent plus and minus two stan- 
dard errors. 

Note 2: The 2011 TIMSS data did not provide standard errors for the 
international average, nor were standard errors calculated for the Top 10 
performing countries averaged score. 
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RECOMMENDATIONS 

The TIMSS assessment is unique in that 
it provides an international perspective 
on how students perform within a broad 
international curriculum. Because Indiana 
state policymakers had the foresight to 
participate in this assessment, education 
stakeholders are able to see how Hoosier 
students are performing at the international 
level. With that said, the assessment only 
provides a snapshot of how education sys- 
tems are functioning in science and math- 
ematics. However, from the data presented 
in this policy brief we offer five recom- 
mendations: 

1. The state of Indiana should capitalize 
on its success as a top-performing edu- 
cation system ready to compete in the 
global economy. 

Students in the 8 th grade performed better 
than their peers across the country in both 
mathematics and science, and compared 
favorably to the Top 10 performing coun- 
tries. In addition, Indiana’s boys performed 
very well in science when compared to 
their peers in the Top 10 performing coun- 
tries. State lawmakers and educators can 
use this information to show that Indiana 
students are globally competitive in the 
fields of math and science. This evidence 
suggests that Indiana’s future workforce 
is well positioned to be a leader in a high- 
tech global economy and the information 
should be used in economic development 
strategies. 

2. The statistically significant gender 
gap between boys and girls should be a 
policy focus. 

If Indiana desires to continue its economic 
competitiveness, then it should provide 
quality education in math and science for 
boys and girls. Yet, Indiana’s results from 
the 2011 NAEP show no gender gap in 
mathematics (Institute of Education Sci- 
ences, 2012a) and a one-point gap favor- 
ing boys in science (Institute of Education 
Sciences, 2012b). The report does not indi- 
cate whether this difference is statistically 
significant, however. Taken together these 
tests paint an inconsistent picture suggest- 
ing that policymakers should seek to clari- 
fy this important issue. 


3. Indiana needs more advanced-level 
students. 

Although Indiana is fairly competitive in 
relation to high-performing students, it ap- 
pears to have a low number of advanced- 
performing students when compared to 
top-performing countries. Closing this 
“achievement gap” (Chien, Spradlin, & 
Plucker, 2007) would improve Indiana’s 
standing and better prepare its students to 
participate in many of the high tech profes- 
sions that contribute to a growing economy 
within Indiana. 

4. The results from TIMSS are only one 
piece of evidence and should be used in 
conjunction with other information to 
inform education policy decisions. 

Education reform has been a hot topic 
within the U.S. and Indiana. The main ar- 
guments for education reform mirror those 
of No Child Left Behind, namely offering 
equal opportunities for all children, creat- 
ing high-quality teachers (Burgess, 2012; 
The Economist , 2012), and developing 
an economically competitive youth (Uni- 
versity of Indianapolis, 2012). The trends 
from TIMSS suggest current reforms have 
produced little change in Indiana’s overall 
performance and international standing. 
Given these results, and the results of the 
2003 TIMSS, the issue does not appear 
to be the underperformance of Indiana 
students or a threat to economic competi- 
tiveness. Rather there are other issues that 
warrant further investigation, such as gen- 
der inequality and a low percentage of ad- 
vanced students. 

5. Finally, Indiana teachers should be 
recognized and congratulated for pro- 
ducing consistently positive results from 
the TIMSS assessment. 

The continued trend of performing above 
the national and world averages, as well 
as producing many high-level-achieving 
students is a testament to the quality of 
teaching that exists in the state of Indiana. 
Hoosier teachers should be praised for 
their efforts and encouraged to maintain 
our educationally competitive place in the 
global economy. 
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